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By 

Prafulla Joglekar 


Abstract 


To assist rational investments in advanced software, a formal, explicit, and multi- 
perspective cost-benefit analysis methodology is proposed. The methodology can be 
implemented through a six-stage process which is described and explained. The current 
practice of cost-benefit analysis at the Kennedy Space Center is reviewed in the light of 
this methodology. The review finds that there is a vicious circle operating. Unsound 
methods lead to unreliable cost-benefit estimates. Unreliable estimates convince 
management that cost-benefit studies should not be taken seriously. Then, given external 
demands for cost-benefit estimates, management encourages software engineers to some 
how come up with the numbers for their projects. Lacking the expertise needed to do a 
proper study, courageous software engineers with vested interests use ad hoc and 
unsound methods to generate some estimates. In turn, these estimates are unreliable, and 
the vicious circle continues. The proposed methodology should help Kennedy Space 
Center to break out of this vicious circle. 
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COST-BENEFITS OF ADVANCED SOFTWARE: 
A REVIEW OF METHODOLOGY USED AT KSC 

By 

Prafulla Joglekar 


Executive Summary 

Advanced software (ASW) investment decisions are multi-stage, varied, complex, 
risky, and controversial. Therefore, we need a systematic methodology to assist rational 
ASW investment decisions. I propose a formal, explicit, and multi-perspective cost- 
benefit analysis (CB A) methodology for this purpose. I outline a number of rich concepts 
and principles of this methodology, and recommend a six-stage process for its 
implementation. In the light of this methodology, my review of the current practice of 
CBAs at KSC finds that the practice Is seriously deficient. 

The basic cause underlying these deficiencies is that we are caught in a vicious 
circle described by the following paragraph: 

At present, CBA studies fail to capture all the relevant concerns. They measure only 
selected costs and benefits using questionable assumptions and unsound methods. As 
a result, the estimated costs and benefits are highly unreliable. Consequently, 
management looks at CBAs not as decision-making tools, but as mere exercises in 
generating numbers for external justification of decisions already made. Thus, 
management does not take CBA studies seriously, and simply leaves the conduct of 
CBAs up to the initiative of the software engineers involved in specific projects, 
without any provision for additional resources and expertise needed for these studies. 
Lacking resources, and the necessary expertise in economic analysis, but with vested 
interests in justifying their projects, courageous software engineers use creative, but 
ad hoc and unsound methods to conduct their CBAs. The resulting cost-benefit 
estimates are highly unreliable, and certainly not worthy of use in any rational 
decision-making. Thus, management's view that CBAs are to be used merely as 
exercises in generating numbers for external justification is reinforced, and so on. 
The vicious circle continues! 

I recommend that at KSC, we should try urgently to break out of this vicious 
circle. The methodology 1 have proposed provides one exit point to break out of this 
circle. The other exit point is a change in management’s perception of what a good 
methodology can do, and its willingness to provide adequate resources and appropriate 

expertise to the conduct of CBAs. 
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Kennedy Space Center 
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COST - BENEFITS OF ADVANCED SOFTWARE: 
A REVIEW OF METHODOLOGY USED AT KSC 


1. Introduction 

Advanced software (ASW) projects are exciting. They keep > us t« the ctmin<r«jP 

of technology they help “^^^"dtp^Se of the brightest and 
development; they promise to capture the Know g v promise to minimize the 

the most experienced personnel in tie space pr ^ ^ id trou bl e shooting in a 

2 = count-down^and t IZTZ 

nadonaTfocuson “**> •"*« " ASW Pt ° j ' C,S ' ha ' 

promise commercial spin-offs. 

As exciting as these pronused be^ 

ensure actual re p " ° p Knowledge-baaed Autonomous Test 
be obtained For Based on hnelligent Computer Operat.ons and 

EL (RUBICON) will no, enable us tTSom and 

» - - 

future compared to the use of these ASW. 

On the other hand, advanced ^of^ 

certain improvements in operations e y un der-utilized. For example, we 

computer systems is that their true po en possible with the 

are nowhere near ^ C detractors of ASW often 

electronic communication p . • but more investment in the 

suggest that what we need is not more habits necessary for 

.raining and in the management of a change m 0 f ASW counter 

developing new ASW. 

In addition, there are a variety of pmj^cS l'taimd'relomcW at 
ASW investment decisions. Given many funding. By their very 

hand, we must decide which ideas to pursue tmd “ f * ° £ rls L „f technical, 

nature, ASW projects take “" P decisions pertaining to an ASW project 

schedule, or operational ure. , b multi-stage decisions requiring a 

— - - * ^ — * 2 
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few examples of the many interesting and challenging issues one has to deal with when 
making ASW investment decisions. 

Some projects, such as the replatfonning of the Shuttle Connector Analysis 
Network (SCAN), seem unavoidable given the obsolescence of the current platform. Yet, 
replatfonning opens several possibilities for enhancements to current SCAN capabilities 
(e.g., LRU trace-through, Automated retest, Wire trace diagnostics, etc.), and total 
project costs depend upon the enhancements we decide to seek. We would be foolish not 
to exploit some of these opportunities for enhancements. However, the larger the set of 
enhancements we seek, the greater would be the project complexity and the consequent 
risk of failure. Thus, the real issue to be decided here seems to be what specific 
enhancements to seek and what not to. 

Some projects, such as the Ground Processing Scheduling System (GPSS), seem 
to deserve continued funding on the basis of their past and measurable successes^ 
However, the issue here may be who should fund it from this point on, and at what level? 
If GPSS’s benefits are clearly demonstrable and the costs of its further development will 
be lower than its future benefits, is it time to spin it off as a commercial venture? Under 
’ this approach, a private firm will have to fund GPSS's further development and share in 
the rewards of its future success. Thus, a larger portion of Code C budget may be 
available to fund other ASW projects which may be too risky for a private (and nsk- 
averse) entrepreneur but quite acceptable to a (risk-neutral) government. On the other 
hand, because of the many complicated legal and political issues involved, attempts to 
commercialize GPSS too soon could actually slow down its development and 

implementation. 

Other ASW projects such as KATE, and RUBICON seem to deserve continued 
funding because they are based on truly visionary technologies. The issue here is whether 
these ASW projects represent a situation of "a solution looking for a problem to solve, 
and whether given our desire for being at the cutting-edge of technology, funding of 
visionary technologies is justified in and for itself. 

Another issue pertaining to KATP and RUBICON seems to be the threshold level 
of funding needed to keep these projects at a reasonably productive pace. For some 
projects, no funding at all may be better than some fundmg below the threshold level. 
One concern is that with the speed at which some ASW projects are proceeding, there 
may be cheaper and better commercial products on the market long before our 
development is complete. Considering that possibility, the question is: Are we simply 
providing taxpayer-funded software development experience to the contractor? 

When funding an ASW project (See Attachment A), we seem to budget for the 
time software engineers would spend on that project. In reality, the project uses many 
other resources in the organization. Computer hardware, and office supplies are the 
obvious examples of these. In addition, there are many hidden costs (hidden until we 
recognize them). For example, to the extent that ASW projects attempt to capture 
corporate knowledge and expertise, they require substantial time and cooperation from 
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various experts. Unless these experts' time is explicitly budgeted for the ASW project, 
project schedule and success may depend on the goodwill of the experts and may even 
risk neglect of the experts' normal duties which may be launch-critical today. Unless all 
relevant costs of an ASW project are uncovered, added-up, and compared with the 
project's likely benefits, one does not know whether that ASW development would be a 

wise idea. 


At the same time, it should be realized that if the experts are not convinced of the 
value of the project, or think that their jobs will be at risk once their expertise is captured, 
software engineers will not succeed in capturing their expertise. In other words, 
successful implementation of an ASW project often requires that each one of the many 
stakeholders of the project should find it cost-beneficial from his/her own perspective. 


In short ASW investment decisions are multi-stage, varied, complex, and risky, 
and their success depends on the cooperation of multiple stakeholders. It is no surprise 
that while there are a few success stories, there are many more instances of project 
failures, long delays, and wasted resources. Thus, most ASW investment decisions seem 
to be controversial. It is therefore imperative that we develop a systematic methodo ogy 
to assist rational ASW investment decisions. 

In Section 2, 1 propose a cost-benefit analysis (CBA) methodology to assist these 
decisions I had hoped to demonstrate the use of this methodology in a couple of actual 
decisio^situalions .Unfortunately, a, KSC the concept of what a CBA -eth^^ can 
do, and where to apply it, seems to be very different than mine. At KSC!, CB As ar 
to justify past decisions, or our preferred choices, to some external constituency. CBAs 
1 not seen as an assistance to decision-making. Indeed, ASW projects that are facing 
serious decision points seem to avoid a systematic CBA. As a result I did not really get a 
chance to demonstrate the use of my methodology. On the other hand as is clear from 
the discussion in Section 2, I did have the opportunity to study several instances of ie 
current practice of CBA at KSC. Attachments A through D present the relevant excerpts 
from the CBAs I studied. In section 3, I review the current practice as a whole 
contrast it with my methodology. Section 4 provides my conclusions and 

recommendations. 
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2. The Proposed Methodology 

Rational decision-makers always assess the costs, benefits, and risks of the 
alternative choices they have. However, this assessment is often informal, implicit, and 
only from a single (the decision-maker's) point of view. I recommend that at KSC the 
assessment of ASW investment alternatives be formal, explicit, and multi-perspective. 
Organizational decision-makers clearly recognize the need for a formal process of 
assessment. An explicit assessment forces us to articulate all underlying assumptions and 
verify their validity. An explicit process is also easier to study, improve over time, and 
pass on from one generation of decision-makers to the next. Many researchers suggest 
that a cost-benefit assessment be "objective." I believe that costs and benefits of an ASW 
lie in the "eye of the beholder." In other words, assessments, by their very nature, depend 
upon one’s point of view, and hence are subjective. Instead of attempting to avoid this 
subjectivity, I recommend that the assessment be from the point of view of each one of 
the major stakeholders of an ASW investment. As I have suggested before, such a multi- 
perspective assessment improves our chances of obtaining full cooperation from all the 
stakeholders, and hence the chances of project success. 

Rational decisions based on such a formal, explicit (therefore well documented), 
and multi-perspective assessment need no further efforts to justify them to our superiors 
or to the general public. 

2.1 Richness of the Methodology 

Formal CBAs have been done for over ninety years now, ever since the 1902 
Harbor Act required that Army Corps of Engineers could build only those water projects 
that could be shown to generate more money than they consumed. Given the language of 
the Harbor Act, the foci of early CBA were on 

(i) justifying a decision already made, and 

(ii) quantifying all costs and benefits in dollar terms. 

In many organizations, these foci continue to prevail even today. However, over 
the years, as CBAs are done in a wide variety of organizations analyzing a wide variety 
of decision situations, the CBA methodology has evolved considerably. In a previous 
publication [1], I have reviewed this evolution, and clarified a number of common 
misunderstandings about what a CBA methodology is, and is not. 

Briefly, by now, we recognize that although a CBA can be used to justify a 
decision already made, its most cost-effective use lies in arriving at the right decision. 
We know that not all cost and benefits can be measured in dollar terms, if they can be 
measured at all. We have developed a variety of techniques such as cost-effectiveness 
analysis (CEA), cost-utility analysis (CUA), and technology assessment (TA) to 
accommodate variables that defy measurement and valuation in dollar terms. More 
importantly, we recognize that rational decisions can be made without forcing a 
quantification of the non-quantifiable, or a prediction of the unpredictable. I see these 
insights and techniques as an integral part of what I call "the CBA methodology." 
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The most fundamental principle of the CB A methodology is to account for (not 
necessarily quantify) all incremental costs and benefits resulting from a decision 
alternative To enable us to do this task properly, the methodology provides a number of 
“principles. For example, i. describes the many different types of costs 
SSbtTi encounter, including: direct and indirect; tangible and mhmgtble; 
fixed and variable; controllable and non-controllable; one-time and recurrent etc. Tbe 
methodology emphasizes the need to account for the opportunity cos, of an act on. The 
prtwe is to count the net benefits we would have : reaped had we taken the best 
alternative action instead of a given action, as a cost of the given action. 

The methodology tells us to pay attention to the cause-effect as well as the multi- 
nroduce^'in^e product relationships as may be present, and to attribute benefits and 
c^s to the cmsL or the producers, as appropriate. It mcorpora.es concepts and toolsm 
.. f the ass() ciated risks and uncertainties. In analyzing a multi-year stream of cost 
^d tnefhs the methodology provides us with technK,ues 

year flows to comparable and consistent unt.s, so that we do not confuse apples 
oranges". In short, the methodology is very rich and insightfu . 


A Clarification of Some Common Misperceptions 


2.2 

Unfortunately in the information systems literature, some scholars have 

—.r " - 
Err^r^ 

emphasize .hat I am recommending a melhodology, not a single technique. 

A methodology includes not only a toolkit, but also an understanding of the 
situations where each r"ls™oaWfit 

methodology [2J^ We recognize m Th. propose methodology 
welcomes a foLaTex^.md rational decision no, to pursue a CB A in such situations. 

The methodology also requires that the scope and the level of detail of a CBA 
study be consistent with the magnitude of the likely ousts of awmng A f ha , 

investment decision, and with the « of „ K be St 

costs $.0,000, when Similarly, . stu dy that 

and the worst choice is only $ , » must be made within a month, 

takes a year to complete will not assist a eci much and take too 

Thus, in my view, a common fear, namely that a CBA will cost too mu 

long, is simply a misperception of the methodology. 
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One widely-held belief is that a CBA is useful only when a project is initially 
approved or disapproved, and it has no role to play in subsequent decisions about annual 
funding levels, etc., particularly so, if an original CBA was not conducted at the time of 
initial project approval. Once die methodology proposed here is in place, there will be no 
reaso/to issume that a CBA with properly defined scope and level of detail cannot ass s 
the current year’s funding decision pertaining to an on-going project, whether an imti 
CBA exists or not. 

Of course, when an initial CBA does exisi. the analysis in sub^querayearsis 
considerably easier. This is so because under my methodology, the initial CBA for an 
ASW pr^ct. incorporating Rapid Pro.otying <RP> . = .ing a . 
development cycle, would include a decision tree analysis (DTA) of the year by y 
alternative possible milestones of accomplishments and subsequent choices. Such a DTA 
speHs^om precisely what to do, once we know which one of the var.ous possible 
milestones actually occurred during the previous year. 

Perhaps the most pervasive misconception of the CBA methodology is that it 
accounts only for the "economic" costs and benefits, and ignores the many ^con^nc 
values we seek. With that misconception, some people even suggest that a CBA has 
rot to play in any government agency, let alone NASA, since government agencies exis 
precisely because market forces fail to provide for certain non-economic societal needs. I 
have shown elsewhere that economists in general, and CBA methodologists in pamcu ar 
have always concerned themselves with the capture of the non-economic values UV The 
methodology I am proposing insists that all values, economic and non-economic, be 

captured, and captured explicitly. When this methodology is ^Trlde 

oreatest contribution may lie in the clarification of the real values at KSC, in such trade 
offs as between obtaining assured launch success using existing (and proven) technology 
md developing ASW for more efficient and effective launch operations in the future. 

2.3 A Process for Implementation 

With this overall framework in mind, I propose that at KSC, we use the six-stage 
process depicted in Figure 1 for assessing various ASW investment alternatives. 

Stage 1 requires that the decision context of a CBA study be articulated 
explicitly That is, we must identify the decision alternatives to be evaluated in 
specific terms as possible. For example, in the SCAN replacing P^ect (See 
Attaclunent A) evaluating the costs and benefits of the total replanning effort does 
i i ’] {_i n n since in face of the obsolescence of the current platfonn, 

not he p any ’ y^ iat we nee( j IS an assessment of the incremental costs and 

benefitsTf^ach' Enhancement sought while replatforming. We must still assess the costs 
and benefits of the basic (no enhancements) replatforming effort, but only to set the base- 
line ^from which the incremental costs, benefits, and risks of an enhancement can be 

assessed. 
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Figure 1 

A Process for Applying CBA Methodology to ASW Investment Decisions 


Stage 1: Context Articulation 

Identify available decision alternatives 
Identify all stakeholders 
Guesstimate upper and lower bounds 


Define Horizon 
List assumptions 


Conduct 
Formal CBA?^ 


Record reasons for 
future 


STOP 


Stage 2: Enumeration 

of these changes: in all pertinent organizations, e.g. : 

' Use of resources "I Sponsoring Department 

J information Input/ output I J Software Development Group 

| NASA Mission Performance r Other directorates 

Contractor Performance J t Contracting Organization 

and Risks (technical, schedule, operational) 

during development of ASW and after implementation 


Can be x 
Measured?^" 

VXyes 


Stage 3: Measurement 

Key issues: Proper base-lines 

joint-use of resources 
co-producers 


Provide clear and 

complete 

description 


Will be 
valued? 


Stage 4: Valuation 

Account for each stakeholder’s point of view 

Resolves issues such as contract terms to determine proper values 

Accommodate alternative valuation metrics 


Provide clear and 

complete 

description 


Stage 5: Adjustments for: ana 136 

time value of money, alte 

risks, 

probabilities of existence of the co-producers 


and Sensitivity analysis with 

alternative assumptions 


Stage 6: Final Assessment 

Combining the valued and the unvalued from multiple perspectives, 
management should: o approvG/ disapprove projects 
o recommend redesign of ASW 
o provide feedback to CBA methodology 
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In addition, in this Context Articulation Stage, we identify all the major 
stakeholders of an ASW project, define the horizon (one year, or five years, etc.) over 
which benefits and costs will be assessed, guesstimate the upper and lower boundson ie 
costs and benefits of each alternative, and make decisions on which alternatives wil 
the subject formal CBA studies, and from which stakeholders points of view. In other 
words, we make a judgment on which CBA studies would be cost-beneficial. 

It is important to define a reasonably long but limited horizon. For example U 
does not help any decision we can make today, if we assess the costs and benefits KATE 
assumkig final completion and implementation of the total KATE vision, which is 
estimated to need $27M in software engineers’ time alone. At the current funding level o 
$300K, it will take ninety years to realize that vision! (See Attachment ). 

In the Context Articulation Stage, we should also begin to compde a lia of 
assumptions underlying our study. In subsequent stages, we should be dd.gent in 

updating this list, as necessary. 

Stage 2 requires the enumeration (or listing) of all the categories of changes 
resulting from an investment in an ASW alternative, both during the development of the 
ASW and after it is operational, but without going beyond the defined horizon. Th 

^^fthrus^f resources including hardware, facilities, labor (both software engineers' 

(ii/i^ormatio^^ quantity , quality, speed and timing, 

(iii) NASA^misTion perfomlce including on-schedule and safe launches, maximum 

Ul pn^u^i S ve 1 u S se°of available resources, being at the cutting edge of technology and 

providing commercial spin-offs, etc., and . . 

(iv) Contractor performance including profitability, productivity, etc. 

We want to enumerate these changes not only in the sponsoring department (e^g 
a vehicle flow manager in the case of GPSS), and the software development group, but 
alL ta the various non-sponsoring but potentially affected directorates and contractors. 

...a before this mav be important in obtaining the necessary cooperation from 
^ without risking a negle* of their norm, 

duties. 

In addition to the above changes, we should also enumerate the technical 
schedule a,“ Lai risks associated with an ASW project. Mso r we should no, 

r ge, t db^htfeeLL l-LLrLsIL of this process to 

Seriappi^ earlier stages requiringLedback and updating from later stages, and vtce 

versa. 

In short Stage 2 ensures tha, we account for all costs, benefits and risks of m 
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measurable ones. After all, making a decision (it. Stage 6) inevitably involves a trade-off 
between the measured and the unmeasured. 

Once the relevant changes are enumerated, it is important to identify those that 
t f anv measurement (e g the quality of information), describe them as clearly an 
^“‘3 decline if they arc still amenable to valuation (petftaps 

through such approaches as the user's willingness to pay). 

When feasible measurement that occurs in Stage 3 is an important preliminary 
valuation* 1 However, even in the case of the measurable, such as the reductron m 
scheduling meeting durations attributable to GPSS (See Attachment C) «e mu« have a 
proper historical base-line measurement, and the ability to project that *«^icin»»wo 

people used to actually attend. 

If nothing else, Stage 3 tells us what data we must begin to collect, so as t0 * rack 
the performance improvements brought about by an ASW. In ™ ^ ^Just 

ASW future, it is b . n P 0 ^ t0 f ^ e f^; ^Tfluenc^the base-line. For' example, 
ex^rnce a Vscheduling past Orbiter flows may also help reduce the scheduling meetrng 
durations necessary for future flows. 

Similarly a reduction in weekend overtune, claimed as a benefit of GPSS (See 
a u rn mflv also be the result of a simple management policy to not approve 

isMe and measure r he incremental contribution of GPSS .0 this reductron m overtune. 

It is important in the measurement stage to identify the many co-producers (i.e. 

. o\ a nmnnsed ASW may need in producing a benefit. For example, o 
necessary conditions) a proposed AoW y v icatF or RUBICON (See 

kate - 

RUBICON will be zero. 

Another issue in the measurement of ASW project befits is 
projects are claiming the same benefits. For example, both KATE and RUBICON may 
be claiming the same reductions in the Firing Room manpower. 
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On the cost-estimation side, a similarly complicating issue is one of the joint use 
of same resources (e.g., the same computer and communications hardware) by many 
different projects. We need to develop a systematic method for identifying the 
incremental changes in these resources brought about by each ASW project. 

Costs are often assumed to be easier to measure than benefits. However, in 
identifying exactly what costs are incremental, there are many issues that need to be 
resolved particularly in the contract management environment at KSC. If contractor 
compensation is based on head-count, will not the savings in direct labor on one task 
(brought about by an ASW) be simply "absorbed" (at least, in terms of their accounting) 
by some other tasks? If demonstrated savings will be accomplished only in future years 
through prudent contract negotiation, such a contract negotiation should be identified as a 
co-producer of those savings. 

In Stage 3, the idea is to measure the changes in resources in their physical units, 
e.g., labor hours, CPU hours, etc. Then in Stage 4, we attempt an explicit valuation of 
these resource changes. Of course, we may deliberately exclude some of the resource 
changes from this valuation. For example, as long as the replatformed SCAN meets the 
desired maximum access time requirements, we may not place an explicit value on the 
system's actual access time. On the other hand, certain changes that could not be 
measured (such as better quality of infonnation) could now be explicitly valued at least 
in subjective terms by the users of that information. This is possible as long as we do not 
insist on valuing everything in dollar terms. Thus, at least until Stage 6, some changes 
may be valued in dollars while others are valued on a "user satisfaction scale" of 1 to 10, 

etc. 


Separation of valuation from measurement is critical in the multi-perspective 
analysis I am proposing. It allows us to recognize that different stakeholders value a 
given change in resources very differently. For example, from a cost-plus-fixed- fee 
contractor's point of view a cost saving has no positive or negative value. For an empire- 
building manager, the reduction in the manpower under his supervision has a negative 
value. If a fixed G&A pool will be collected by the contractor by the end of the year, 
regardless of the direct labor hours involved, should not G&A be left out of the rate 
NASA uses to value each labor hour saved? The proper labor rates to use in Attachments 
B, C, and D can be arrived at, only when issues of this sort are resolved. 

For many other resources such as computer hardware or office facilities, market 
prices are commonly seen as an "objective" source of value. However, economists point 
out that market prices are not value-free; they derive from a particular income 
distribution and from existing institutional and legal arrangements. As such, at times it is 
necessary to adjust market prices to reflect specific stakeholders values. For certain 
benefits, sucli as the improved quality of decisions supported by an ASW, market prices 
may not be available and valuation must be imputed from the relevant stakeholder's 
beliefs, attitudes, and preferences. 
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Clearly, a number of assumptions are required in this valuation stage, and we 
must not forget to update our list of explicit assumptions. Sometimes, during valu 
we realize that someLgs we had originally decided not to measure can and need to be 
measured. Thus, there may be a feedback from this stage to Stage 

In Stage 5, the explicit values must be adjusied for the timing and uncertainty of 
their occurrence It is in tins Adjustment Stage that we must also adjust for he 
probabilities of existence of the co-producers of out benefits. These adjustments often 

^^r^sensUivity «.e.. what-iO analysis considetmg abemauve 
vies fotThe various assumptions, e. g.. alternative discount rates, al.emat.ve tunmgs of 
occurrence of particular events. 

At the conclusion of Stage 5, the analyst's task is complete. In Stage 6 , the 
decision-maker(s) must consider the valued and die “-valued mge.het tan each 
stakeholders point of view to arrive at the final assessment of an ASW alternative. 
Sometimes this Final Assessment Stage may provide a clear decision regar g 

we may have to repeat the entire process beginning with Stage 1. 


Implementation Requirements and Advantages 


2.4 

From the many analytical issues I have identified, i, shoo'd tade».hmrte 
conduct of this methodology cannot be left to the software 

“ rSTS “2w"S£,“ « .ia — t— — - 

valuation issues may be already resolved. 

I think that an investment in this methodology will pay back many times over 
I think ha an mvestme suggested in the foregoing discussion, the use 

) No rlhZl efforts needed to justify the decisions to exremal bod.es, 

^ ^T^rojec, from the multiple 

improving the probabUities of existence of those co-producers. 
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3. A Review of the Current CBA Practice at KSC 

Before I say anything else, I must say that I appreciate the willingness of the 
authors of the CBAs in attachments A tlirough D to subject their studies to a 
methodological review. Given that they had no background or training in the relevant 
philosophical and economic issues, I admire their creativity and courage in authoring 
these studies. I mean no harm or insult to these authors when I point out the conceptual 
errors in their methods. 1 particularly admire them for recognizing, on their own, that 
most of their numbers were simply wild guesses, and that the margin of error in t eir 
estimates was perhaps very large. I am most encouraged to find that these authors are 
highly interested in obtaining the necessary background, and in developing a better 
methodology for the future. 

In Section 2, 1 have already commented on many specific conceptual issues in the 
studies represented in Attachments A to D. I will be happy to provide additional detailed 
comments and suggestions to the authors, if they so desire. However, here I want to 
review the overall practice of CBAs at KSC. In the light of my proposed methodology, 
we can observe many deficiencies in the current practice. However, two important 
deficiencies seem to be the root causes of the rest of them. 

First CBAs are not done to actively assist the decisions at hand. Instead, they 
seem to be produced for public relations (i.e., justification of past decisions), or 
documentation requirements (in the justification of a preferred decision). In project 
review meetings 1 observed, CBAs were often introduced casually with phrases such as 
’’now let us see where we are going with our numbers." In other words, they are given 
little credibility, and practically no scrutiny. 

Indeed at KSC I have observed instances where managers facing complex 
problems deliberately avoided CBAs. I believe that this practice is based on the many 
misperceptions of what a CBA is, and how it can assist decision-making, discussed 
earlier I hope this report helps correct that misperception. Al the same time, as I will 
explain in a minute, given the cuiren, state of CBA practice a. KSC, these managers were 

fully justified in avoiding CBAs. 

Second, the conduct of CBAs is left to the initiative of software engineers who 
have little background, training, or assistance in the pertinent methodology. Thus, each 
study seems ad hoc, developing its own methods and concepts. Indeed one engineer 
suggested that it was KSC's standard operating procedure "to build a brand new road 
every time we want to go to Orlando! 

Each one of the available studies seems to violate one or more of the fundamental 
principles of the CBA methodology. None of the studies I examined tried to capture^ 
die costs and benefits, as is required by the methodology. None of them made all of their 
underlying assumptions explicit, or estimate probabilities that the explicit assumptions 
will be valid Most studies did not seem to use proper base-lmes or proper projection 
methodsbi the measurement of their costs and benefits. TTiey faded to separate 
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measurement from valuation, and to address the many issues of vidua , on from the 
perspective of the multiple stakeholders. Even the more commonly un ^ stood P “ , ' C “ 
of ,|,e CBA methodology, such as adjusttog for time value of money m a multi-year 
stream of costs and benefits, were not used in the CBAs at KSC. 

In short, the current practice is seriously deficient. 

Sneaking as a professor, I am sorry, but I must assign an F grade to this practice. 
At the same time I must add that despite this team grade, most individuals who are 

3.1 The Vicious Circle 

As I think about the two root causes of deficiencies together I have come to 
realize that we are caught in a vicious circle which can be described as below: 

. Available CBA studies measure only selected (not ' han «“ br ^« h ' a ^“‘ b ^' 
rievelooment and implementation of a given ASW« » 

stakeholder s separate) point of view. Values are not adjusted for their probability or 
dmnig of occurrence Sensitivity analysis is no, done. In short, many principles of the 
CBA methodology are violated. 

e rtus of the CBA studies is primarily on the £ 

of senior managers, the values of the contractors, etc., are no, captured by 
analysis. 


Then, 


n p»„cp CBA s do not capture and address the real values and issues, and because the 
decisions already made. 


Thus, 


Management allocates few resources, and leaves the conduct of CBAs up to the 
initiative of the software engineers involved in specific proje . 


Next, 
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• Lacking resources, and the necessary expertise in economic analysis, but with vested 
interests in justifying their projects, courageous software engineers use creative, but 
ad hoc and unsound, methods to conduct their CBAs. 

But this results exactly in the situation described in the starting bullet of this process, and 
the vicious circle continues! 

Figure 2 depicts this vicious circle graphically. 
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Figure 2. 

The Vicious Circle 
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4. Conclusion and Recommendations 

I have argued that ASW investment decisions are multi-stage, varied, complex, 
risky, and controversial. Therefore, we need a systematic methodology to assist rational 
ASW investment decisions. I proposed a fonnal, explicit, and multi-perspective cost- 
benefit analysis (CBA) methodology for this purpose. I outlined a number of rich 
concepts and principles of this methodology, and described a six-stage process for its 
implementation. In the light of this methodology, we reviewed the current practice of 
CBAs at KSC. 

Although I have concluded that current practice is seriously deficient, I believe 
that most NASA employees already knew that, and many are looking forward to 
improving that practice. I think my principal contribution is the identification of the 
vicious circle we are in, and consequently, my primary recommendation is. 

Break out of that vicious circle. 


The methodology I have proposed provides one exit point to break out of this 
circle. The other exit point is a change in management's perception of what a good 
methodology can do, and its willingness to provide adequate resources and appropriate 
expertise to the conduct of CBAs. 
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attachment a a-i 


5 . 


5.1 


Excerpts from a CBA of SCAN Replatforming 

coati B*n*flt* 

Benefit* 


Ben* fit * SCAN are difficult to 

V, fit* for the repl at forming of SCAN lth roan dated 

The benefits . are primarily associate the OSF 

quantify ^ ca ““ DH .' V Tha planned »lgr*tt<in ourc *„t SCAM 

£££".«««- «*» T: W OilioX obaoiatawnlcn-ans 

components (r*« u ~! ' However, with the necess* j 

t^t tepi.tfotj.in9 V rt tnit Y «iu r b e” KT-W 

f!q„ifioint i»ptove»ent. Penefita to te 

SSui- •*«« ~ ' w . to par( otm 

aS “ 

no ?*rbago language. 

required by the m da ta 

^£\ 0 

Stminatton m'Jm.S^tcuUty a^oon 

8 -» fi^U at. rapa tta 

Elimination of unua.d r ?S 8 *°need. of the uaat 

that are mor 

community * 


5 2 gufc-BrMKrtQ* 0 * be 

con junction*wit b °t b a Wt data avail-*- 


in 


5.2.1 


Development Coet Mtimetee 


i qr*AN |d© s t 

- ‘*;^ op r c t.^.A e ^:^J^rirt^‘ 

^seated by CASE Socumeniation^^ach of ^®«® n at ^ 3 

Design, * "?th manpower estimates baseu 

described below with time, 

information available a 


are 

beat 
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A- 2 


ensure system secure y, 

for design. . d requirements from 

The Dealgo Stage will tate the^ f^ y ^f^chnlcS 
the Analyst 3 Stage an £ la , given the £ o£ 

I^iir: ind ptavioua — 

aUtO “ aCi0n- t ' HiU code and teat 

The Build stage tali ^ depend on the 

ssssE -in*. .. - 

concurrent build stage. the described 

The current 

" l8V ^ eot detalla. 
tnAivoid smwp^^y / 

3ee “ eEMDIX *' 3 Ma ° P ,„-davs, -XU- 


Analysis Stage 


Design Stage 


Build Stage 


Documentation Stage 


Total man-days i 
Total man-weeks i 
calendar weeks; 

Total man-days i 
Total man— weeks i 
Calendar weeks. 

Total man-days; 
Total man-weeks i 
calendar weeks; 

Total man-days i 
Total man-weeks; 
Calendar weeks; 


_63£- 




JU- 



, thl8 analyst. It an Ration that.the 
me primary “"P^ ° not b , achievable per manning 

:aplat forming » y oucc ent "“‘“L. ^Lution d »“ l * £ « 

jchedule at n February H _ olanned schedule of a 

levels indicate that a f TO meet the planned ec a s 

jgg^completion^date would regulre tnoreas 
follows; 
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ataoa 

Analysis 

Design 

Build-Doc 


Required 

HanpQWft* 


5.6 

4.6 
8.2 


HI* * 4 Propoaad Development *“ ha ^“Jt ma ted 

3aa *PP» nd ** scheduling implication o£ -the 

Milestones, to* 
manpower requirements. 

522 implementation estimates 

sirs?? ^By^hS € T t 9Ss- 

Sjacu.aeS in aaction 4 . 3 . 3 . 


5 . 2.3 


Running Coat Ratimetea 


UOiw pitvaw-rw. 

Running coat ^^“tiolTof^theVo”^ “aka 

Eris. sst.^jr'^* -* be ^ co 

^UaCiy a^port tha plannad R0BM3. 

wniia .U - - SSI * ~ 

daoraaaa in Sparatlonai tunning ooata. 


5,3 prin t- I 'mt"-* 1 * »n»lv «U 


5,3 virtua l ly impoaaible 

i «i a of coats versus benefits is V OUHlt , er of 

Tha aMly* 1 ’ t of SCRH piat£ot»tng^ software 

caaaona? The ^^^0 'l* JJi. 

quantifiable* and aome of tha t hr'f ^ tf o tn i ng wi» be 

The beat that “^tad b? tha eatimatea that 

expensive as perspective, it mua SCAN system to 

k it e has^a?en more than five J^^Jonaiity, i^ lud ^ *coats 
ac,UeVd t ed t with U app?orimateiy 300 Ptobiem^eports . that SCAN is 

associated w n PP dat imates are a v* 9 commitment 

JSirT ‘coipiarayatan and thay ai.o tR. firjt 

to flald l «P‘f/ OC :£. 0 2& ^»t wiii continue the 

time end r f ad y„,_ gnaineering worKload. 
reduce the System engineer* ? 
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attachment D b-1 


. flil . KnowledM Dw«l Aulonomous 

Costa Uenefitt Analysis <" r '{** eXmcTk ATE) 


IN1R KATE is a |ool for health monltorinf 

which ‘.TulTs by'sV.rf V ; Vr !oe 

forecasts, “ vc ‘*V“o“i,dopMnl VnO dcplovmeot of mtelhgen. | Pf“« n eeded for a 

riSr»»“ “j skxstjs. f. ; r. i. «o tsfsss*^ 

&3&ggSSa££&E&g!s. 

rrrzi - - - — 

%%££&' tfl » tKin t «onT.°AL. Pino, Room 

applications. ^ ^ esUmatcs for 

*■ s 

3 Since at the current time only j«j 

. . n_-«:e..al«r Shuttle SVSlCin Is < 


economy of scale rom on # Pftrt icul»r ShulUe system is defined as 

„ A n««^° 0 f . , S^^"„ t nS'"a>o« (FDs) associated wiU. that system. 

the number hr/FD ThU resumption is ba 

}. A measure of ^ 1 s ^tAT^^‘“*" ,nn,M (U *“ ‘" C ' UdCS ** “ 
6. The labor rate is define4 M 40.00 1*.. ( .Pi— * 


based 

time 
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ORIGINAL Pm OF !S 

Of POOR QUALITY 



SUMMARY OF ANALYSIS 
IMMEDIATE BENEFITS ( < 3 yrs) 

1 . KATE can draw conclusions on integration 

analysis of measurement mpui 150 Sw per vcluclc per flow may be realized due lo 

pc ycc. wou,U ct,u.,c ,o 

$ 900,000 per year. 

POTENTIAL BENEFITS (>3yrs) 

2. KATE represents one ^ analytical ' savfngs arc m 

changing the knowledge base ^ 1 . t j ic knowledge base for each class of 

rcdUCCd ^ cycle list savings arl in sustaining engineering. 

sKX^me reasoning software is used for all subsystems. 

2 | Cast Pupf nilillires 

6 

@ $ 30 Million, 
diagnostic capabilities. 

2 2 Filimnifii Cost Saving* 

Susuining «M* ; S“ * ^ ^ S,5Km> °" ‘ ^ 

arc estimated to be @ $ 5.67 Mituon. 

Note a... aas — r^S?— of goal 

less than the esutnat^ engineering cost savings for 

~ - w 

<§> $ 2.33 Million . 


B-2 
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• es may bo o„ . yo^.y ta* * «—" d '° * 

The total costs savings that may 
@ $ 7*95 Million per year. 

Other Benefit* . d la ana lysis or as a 

. , for oUicr oonso.es based on 0.0 .«* «“> on * 

, «txe of launch team tor auuwv 


onic'- 'iM c 
°* *X)fl Q 


K5 

u/»t/ry 


t 
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. . ,„ H takp into account the uncertainty 


* 
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4> 06 .K. « 5 f.» 4.963 Klines 


c rnTAI KATE O* APPLICATION 

ESTIMATED SIZE OF TOTAL KATt 


259 


goal 


sustaining engineering 


B-6 


ESTIMATES 

7 mUION UNES OF GOAL CODE <•> 
,00 S/W ENGINEERS!®) 
usoc LABOR RATE “ 40 * /tir 

ONE MAN YEAR’ 2000 hr 

sustaining eNomeemNO o 20 00 hrsX 

COSTS CST IRATE 


* ft GOO ooo $/Yr 

40 $/!>(■ X 100 - * 8 , 000 , 


flOQfi.QRlH 
7,000,000 lines 


,.143 Vlloe 


KATE ESTIMATE 
4.»V» .W* 


* 5,670,000 $/Yr 


POTENTIAL NET SAVIN 0 00 $/YR 

. 5 6 70,000 $/YR 51 2 ‘ 33 °’ 

8,000,000 $/YR 


I 
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ATTACHMENT c 


C-1 


Excerpts from ft CPA of GPSS 


“rT—. -rrsaswss - S3SSH2 

iSSSM 

schedules. 


uiilUaii the GP ^. ma „ v duilno » 16 wee * ^ n , for leclmlclan support This r ^ abla |0 

r — io ‘ acas,, " u ol 

sssr jjsi 1 “ e '""" 

conflicts- * w '° 


;onlllct&- ..laiart at $528,809. 

, ftT o 60 oV-102 are esllmal®® el * 

Total cost saving# l« ST3 60 ' OV 
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ATTACHMENT D D-l 


RUBICON COAT AHM«V810 ATUPY 


evaluate tbe |»UO*C°M co,»o«pt ro« 1 • too into ccms i aa 

1 1 ‘ir U.2°o? «?sr U proviso a coat pay back 
shuttle o|>oretio«q. 

K^^rTelop^nt will continue l« th, direction scribe, 

,,,o following itowo — Planned for Implementation in r»«.^ 

° thl8 
O TI.O MOT CLIPS portion will be Incorporated to r«n un er 

UUUICON 1 nmnm (l8V , l0 p.d by nocHweli will b. 

° ”r?u.‘5*“i5. upa koyntrokeel . 

O nuuicOM must be converted to run under MOTIP. 

8VBtera 

. f _ r »i lowing vehicle monitoring trom * 

0. The wanageawnt loouoa f firing Boowl will be worke . 

remote location «ouca „ on . „ ln9 le network 

c. The n a » 0mi0 “ 8 °'' d o 8Hd a valldated. Tide la currently echeduled to 
be ^ complete^ around the .»ld-19« tl»<r— . 

“• vz 11 

maintained end exported, 


262 


OmGiNAL PAGE IS 
Of POOR QUALITY 


D-2 


nUPICOti COST AWMiYfllO flTUOY 


JI T, POfll TTVB r^ flT flAVIMQfl 
A, optimization at manpower resource# 

pencript loni J^^^ito^aU^ ylhlcS^ 

Kru u ?..TS»u»? p "^'Hr, OKl “ Mly u5% of 

ti» time the vehicle lo powered up. 

,!l,t on 

Second ShW? -Vl onUilrd ““ 

to 10 people a witablllty by Dl»S hardware engineers 
method for uur ^*i*"*oort when ohuttle budget 
co continue vehicle » Pg ar c oe ayatem engineers in 
cutbacks Impact rhai engineers eupportlng 

che croup. Jha number of en«‘»®«‘J 0 people per 

sr'i tio^ri^ei St *>p‘e » -» » u " quc 

Impact iinf vehicle coating. 


n runld opening unnaceaaary lPR*e 

' neacrlptioni S55SS/SK.- 

date. With no Ipn {5 l ft _Jlt® t to y l} o researched long 
IPH*o can be ^®”® d t ? A t V the problem waa eeen before 
enough to f / Edition . The engineer must 

;i2n Slow^hi m ea an eaplalnad condition. 


Eut Savings i 


Of * 


'**ag‘* 

•‘-fV 


The re°hao*been an average Spened^er'yea^ leveraged 

over the last 12 yearjl ; ^ potentially 

phone 16 that HUBICOH c blenUJ we re either 

avoided approximately 9. J* ® ^ datft in the IhJOiCON 
addrenned u.iderotand the problem and 

databaue c ? ulU J™Xln \\ X q ^eotlemted coot to open and 
avoid opening an figure doea not include 

UruZ raguiced 2 to°lnvoatlgato tha problem.. 


o. ~-u. •»«•- •> - *7”“: ri- .. . T ., 

SS^^rswssara.”-^ 
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be rationalized ao being OK to launch given certain 
conditions ure met) that would allow the count to 
“.2, M quickly you could preclude ... unneceqaary 

launch scrub. 

Eat saving.. A wt.li.num «l ..UUon o.vlng. wold be r.alU.d. 

Tradeoff between OOM. y.r... RUBICON maintenance for CM By.t.» 
dlupUy monitor inu ♦ 

IlKucrlptiot" 

^SpaEwKSbM sst 

system monitor software. 

eat saving., No coat saving, but no .ddltlo„.X co.t Incurred. 

B. improved training for new hire* 

Description. When new 'l^Sh^llcSnSiCtra^ WUh t^Sd 

can bo tra “‘ ea ?b« SpS ny»tem works. Faiiurea 

t^teat ^-^ir.epor^e b |r- 
tlli^CHruo^tnu^av^dln. ech. doling conflict, and 
reducing “he Impact on other ayetema. 

On the software development .id., ^»^^^" u age . 
c ime frame. 

est Savings, saving, is.bard to determine but has the potential 
Eat odvingu t ^ 0 i g i«lf leant amount. 

„ offlc./Plrlng Room tool to raduo. the time it takes to find 
historical Information. 

. . T pn/pn hiotorical data and the PHW muat often be 

Description, « co „ t anything from general 

..anagement gueotlona/concer,.. to^troub^e.boot l. 

problomo . by oearc ung a q reduce the manpower 

r,u^2d l "%iotirthe n n V ece..aru information. 

F.„t Saving., $‘ 2 ;“““^°^ sy.tem engineer, .pending .5 hour, per 
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e«c 


, , RUBICON 

a a 20 mlnuteu. 

_ ha fl and 0®pabUiti«» tov CCH8 *♦ 

,.„. <.»■•<- '" •»,k;2S“.3S *8- , 

5? sKBT 

could converoely, te ^ lwl T? ?»•*»•"«> wUtaKeo 

^mvoi^r.: 5- -chs , e K Blde 

ill reauUe * lwaoflive il ear, oro1ecto will h Q 
3t Savings t -w ^P^cnas to -Y*- 

health monitoring. 

— ,,aw #ppt °‘ c,,M #nd ~ pBhUUl ” for TL 

SEw" uUU«J 

W®? in <■» 

new environment. algnlfl cant 

... ...i— 

s‘;tnnw-c«?S3s t 

«“ ^.SKSi^Kr. — w-" 6 

"wditicntlons. ure»“V 

r ime/COStU • 


G9K®K£i 
of P»dOR QUAl.i Tv 
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RUBICON COST ANAhYfllfl STUDY 


i y. MKQAT TVP COAT flAVlMOfl 


A. 


Dual maintenance of aoAb displays end Rubicon Bystem Monitor. 


Descriptions 


Purina the timeframe from October 1993 until ccms 
2 lu operational « there will be maintenance 
required on both the GOAD noftware ae well ae the 
Syutem monitor portion of RUBICON. 


Coat Impact! 


wlli° require approximately 50% manhour increase on 
mumUtoiy Uooiun center change packages. There Y? re 
26 mandatory change drlvere (that impacted the 12 
goad display programs that DM5S can replace) Qvei 
the past 2 yearo for a total of approximately 550 
manhours. Thin figure doea not include the bcc 
monitor , 


B. Additional maintenance required to maintain Chips rules. 
Descriptions 


An additional .5 engineer would be f? fl 

maintain the expert system portion. (Note i ^‘ie 
includes maintenance on the bCC, HOT and any other 

Chips module.) 


Coot impact! 


coot^io^more than absorbed in the reduction- of the 
number of system engineers required. 


c. Additional maintenance required for database. 


Description! 


Coot Impact! 


There will be a small increaee in manpower required 
to maintain the database. However, automation 
routine owl 1 l make this task a simple procedure. 
Database routines can be run while other tasks are 
performed. 

S2 100/yenr (0 flowo/year x 8 hr/flow x $33/hr) 
it will t2ke an estimated 0 hours per flow to 
maintain the database. With documents on line there 
would be less need to manually update the j P a P e J[ 
version of the documents, A reduction in document 
distribution can also be realized as well as 
reducing the amount of paper used. 


D. Additional work required to set up additional CM tracking 
procedures « 
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Doucript Ion i 


Coot Impact i 


The initial development of 

' .Minimal impact and i« already in «< l J"f " OUA 
he a one time impact that coul< J. aU C 

proceuaeo) require periodic modification. 

that are being developed anyway. I 


office hardware maintenance eoete 
oeucrlptlont 


Coot Impact! 


froin tliwo to tinw« 

covero a period of appranlmately J yearol . 
are in work to fold the maintenance of theee 

fe-SSiK as 

Siact wot U not known «t thin tlma. 


clips validation. 

Dooctlptiom 

co«t Impact) exact coat to not known at thin tima. 
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